[SPARK-56451][DOCS] Document how SDP datasets are stored and refreshed#55277
Open
moomindani wants to merge 1 commit intoapache:masterfrom
Open
[SPARK-56451][DOCS] Document how SDP datasets are stored and refreshed#55277moomindani wants to merge 1 commit intoapache:masterfrom
moomindani wants to merge 1 commit intoapache:masterfrom
Conversation
moomindani
force-pushed
the
sdp-doc-storage-refresh
branch
from
5763f95 to
723afa3
Compare
dongjoon-hyun
requested changes
Apr 10, 2026
Member
There was a problem hiding this comment.
Hi, @moomindani .
Apache Spark community uses JIRA IDs for bug tracking. Your PR title is wrong.
SPARK-55276 is SPARK-55276 Upgrade scala-maven-plugin to 4.9.9.
moomindani
changed the title
Add a new section to the Spark Declarative Pipelines programming guide that explains the storage and refresh mechanics, including: - Default table format and how to specify a different format - How materialized views are refreshed (full recomputation via TRUNCATE + append) - How streaming tables are refreshed (incremental processing with checkpoints) - Full refresh behavior for both dataset types
moomindani
force-pushed
the
sdp-doc-storage-refresh
branch
from
723afa3 to
4c09d8f
Compare
Contributor
Author
|
Thank you for pointing that out, @dongjoon-hyun. I've updated the PR title and commit message to use the correct JIRA ID: SPARK-56451. The GitHub issue has been closed. |
jaceklaskowski
approved these changes
Apr 11, 2026
Contributor
jaceklaskowski
left a comment
jaceklaskowski
left a comment
There was a problem hiding this comment.
LGTM (with some tiny changes)
| </div> | ||
| </div> | ||
|
|
||
| SDP itself does not restrict which table formats can be used. However, the table format must be supported by the configured catalog. For example, a Delta catalog only supports Delta tables, while the default session catalog supports Parquet, ORC, and other built-in formats. |
Contributor
There was a problem hiding this comment.
What's a "catalog" here? Table formats are set up via packages on command line when Spark Connect server's started.
|
|
||
| This means that every refresh is a **full recomputation** - there is no incremental or differential update. For tables with large amounts of data, be aware that each pipeline run will reprocess the entire dataset. | ||
|
|
||
| Because of this mechanism, the materialized view's underlying table format must support the `TRUNCATE TABLE` operation. |
Contributor
There was a problem hiding this comment.
Suggested change
| Because of this mechanism, the materialized view's underlying table format must support the `TRUNCATE TABLE` operation. | |
| Because of this mechanism, the materialized view's underlying table format must support the `TRUNCATE TABLE` operation (e.g., Delta Lake). |
| 2. New data is appended to the existing table data. | ||
| 3. A checkpoint tracks the processing progress so subsequent runs resume from where the last run left off. | ||
|
|
||
| Streaming tables require a checkpoint directory on a Hadoop-compatible file system (e.g., HDFS, Amazon S3, Azure ADLS Gen2, Google Cloud Storage, or local file system). The checkpoint directory is configured via the `storage` field in the pipeline spec file. |
Contributor
There was a problem hiding this comment.
Suggested change
| Streaming tables require a checkpoint directory on a Hadoop-compatible file system (e.g., HDFS, Amazon S3, Azure ADLS Gen2, Google Cloud Storage, or local file system). The checkpoint directory is configured via the `storage` field in the pipeline spec file. | |
| Streaming tables require a checkpoint directory on a Hadoop-compatible file system (e.g., local file system, HDFS, Amazon S3, Azure ADLS Gen2, Google Cloud Storage). The checkpoint directory is configured via the `storage` field in the pipeline spec file. |
|
|
||
| ### Full Refresh | ||
|
|
||
| You can force a full refresh of specific datasets or the entire pipeline using the `--full-refresh` or `--full-refresh-all` CLI options. A full refresh: |
Contributor
There was a problem hiding this comment.
Suggested change
| You can force a full refresh of specific datasets or the entire pipeline using the `--full-refresh` or `--full-refresh-all` CLI options. A full refresh: | |
| You can force a full refresh of specific datasets or the entire pipeline using the `--full-refresh` or `--full-refresh-all` CLI options, respectively. A full refresh: |
Contributor
|
Please add |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Add a new "How Datasets are Stored and Refreshed" section to the Spark Declarative Pipelines programming guide. This section covers:
parquetviaspark.sql.sources.default) and how to specify a different format with Python and SQL examplesWhy are the changes needed?
The current programming guide explains how to define datasets but does not explain how they are stored or refreshed. Users need to understand:
--full-refreshactually does for each dataset typeWithout this information, users cannot make informed decisions about table formats, storage configurations, or pipeline performance.
Does this PR introduce any user-facing change?
No. Documentation only.
How was this patch tested?
Documentation change only. Verified the content is accurate by reading the SDP implementation (
DatasetManager.scala,FlowExecution.scala).Was this patch authored or co-authored using generative AI tooling?
Yes.